{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GfiQHWweOb-C"
   },
   "source": [
    "# TextAttack on Keras Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gPamLGzzdcha"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/QData/TextAttack/blob/master/docs/2notebook/Example_6_Keras.ipynb)\n",
    "\n",
    "[![View Source on GitHub](https://img.shields.io/badge/github-view%20source-black.svg)](https://github.com/QData/TextAttack/blob/master/docs/2notebook/Example_6_Keras.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wv850rJPPE99"
   },
   "source": [
    "Please remember to run  **pip3 install textattack[tensorflow]**  in your notebook enviroment before the following codes:\n",
    "\n",
    "## This notebook runs textattack on a trained keras model: \n",
    "\n",
    "## Training\n",
    "\n",
    "The code below trains a basic neural network on a series of movie reviews from the IMDB dataset, loaded using Tensorflow's datasets module. Each review is encoded as a sequence of tokens corresponding to a word's index in the vocabulary. Class labels are provided, denoting a positive or negative sentiment. \n",
    "\n",
    "See [here](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb/load_data) for more information on the IMDB dataset. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Oru-kBljwyqd",
    "outputId": "7c703b51-1fd5-47cc-c1e4-c0fba3b9bdfa"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import keras\n",
    "import numpy as np\n",
    "from keras.utils import to_categorical\n",
    "from textattack.models.wrappers import ModelWrapper\n",
    "from textattack.datasets import HuggingFaceDataset\n",
    "from textattack.attack_recipes import PWWSRen2019\n",
    "\n",
    "import numpy as np\n",
    "from keras.utils import to_categorical\n",
    "from keras import models\n",
    "from keras import layers\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import Flatten\n",
    "from keras.layers import Dropout\n",
    "\n",
    "from nltk.tokenize import word_tokenize, RegexpTokenizer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QUT675TAVVle"
   },
   "source": [
    "Below, we load the IMDB dataset from Tensorflow and transform it for our classifier, using a Bag-of-Words format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6rR709EZvO6O",
    "outputId": "9d51ee80-2352-47b7-c864-75cfca90024a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
      "1646592/1641221 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "\n",
    "NUM_WORDS = 1000\n",
    "\n",
    "(x_train_tokens, y_train), (x_test_tokens, y_test) = tf.keras.datasets.imdb.load_data(\n",
    "    path=\"imdb.npz\",\n",
    "    num_words=NUM_WORDS,\n",
    "    skip_top=0,\n",
    "    maxlen=None,\n",
    "    seed=113,\n",
    "    start_char=1,\n",
    "    oov_char=2,\n",
    "    index_from=3\n",
    ")\n",
    "\n",
    "def transform(x):\n",
    "  x_transform = []\n",
    "  for i, word_indices in enumerate(x):\n",
    "    BoW_array = np.zeros((NUM_WORDS,))\n",
    "    for index in word_indices:\n",
    "      if index < len(BoW_array):\n",
    "        BoW_array[index] += 1\n",
    "    x_transform.append(BoW_array)\n",
    "  return np.array(x_transform)\n",
    "    \n",
    "\n",
    "index = int(0.9 * len(x_train_tokens))\n",
    "x_train = transform(x_train_tokens)[:index]\n",
    "x_test = transform(x_test_tokens)[index:]\n",
    "y_train = np.array(y_train[:index])\n",
    "y_test = np.array(y_test[index:])\n",
    "y_train = to_categorical(y_train)\n",
    "y_test = to_categorical(y_test)\n",
    "\n",
    "vocabulary = tf.keras.datasets.imdb.get_word_index(\n",
    "    path='imdb_word_index.json'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7DpUFR0fVmzz"
   },
   "source": [
    "With our data successfully loaded, we can now design and trained our model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YjLVCp3Z0SUj",
    "outputId": "ba7eda9d-7f74-4cc2-d366-6c13329d96e3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/18\n",
      "44/44 [==============================] - 0s 9ms/step - loss: 0.9584 - accuracy: 0.4987 - val_loss: 0.7314 - val_accuracy: 0.5056\n",
      "Epoch 2/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.9078 - accuracy: 0.5064 - val_loss: 0.7149 - val_accuracy: 0.5332\n",
      "Epoch 3/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.8743 - accuracy: 0.5264 - val_loss: 0.7000 - val_accuracy: 0.5600\n",
      "Epoch 4/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.8534 - accuracy: 0.5385 - val_loss: 0.6840 - val_accuracy: 0.5904\n",
      "Epoch 5/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.8329 - accuracy: 0.5564 - val_loss: 0.6754 - val_accuracy: 0.6064\n",
      "Epoch 6/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.8168 - accuracy: 0.5615 - val_loss: 0.6637 - val_accuracy: 0.6348\n",
      "Epoch 7/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.7942 - accuracy: 0.5767 - val_loss: 0.6568 - val_accuracy: 0.6460\n",
      "Epoch 8/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.7798 - accuracy: 0.5895 - val_loss: 0.6464 - val_accuracy: 0.6632\n",
      "Epoch 9/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.7624 - accuracy: 0.6000 - val_loss: 0.6357 - val_accuracy: 0.6772\n",
      "Epoch 10/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.7523 - accuracy: 0.6096 - val_loss: 0.6275 - val_accuracy: 0.6932\n",
      "Epoch 11/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.7391 - accuracy: 0.6185 - val_loss: 0.6196 - val_accuracy: 0.6996\n",
      "Epoch 12/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.7265 - accuracy: 0.6324 - val_loss: 0.6126 - val_accuracy: 0.7076\n",
      "Epoch 13/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.7140 - accuracy: 0.6408 - val_loss: 0.6047 - val_accuracy: 0.7196\n",
      "Epoch 14/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.7042 - accuracy: 0.6476 - val_loss: 0.5981 - val_accuracy: 0.7268\n",
      "Epoch 15/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.6944 - accuracy: 0.6586 - val_loss: 0.5906 - val_accuracy: 0.7340\n",
      "Epoch 16/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.6798 - accuracy: 0.6677 - val_loss: 0.5826 - val_accuracy: 0.7432\n",
      "Epoch 17/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.6702 - accuracy: 0.6766 - val_loss: 0.5741 - val_accuracy: 0.7524\n",
      "Epoch 18/18\n",
      "44/44 [==============================] - 0s 6ms/step - loss: 0.6643 - accuracy: 0.6834 - val_loss: 0.5667 - val_accuracy: 0.7580\n",
      "{'loss': [0.9584308862686157, 0.9078119993209839, 0.8743314146995544, 0.8533967733383179, 0.8329190015792847, 0.816802442073822, 0.7941828966140747, 0.7797670960426331, 0.7623777985572815, 0.7523201107978821, 0.7390732765197754, 0.7265127897262573, 0.714047372341156, 0.7041717767715454, 0.6944125294685364, 0.6798228025436401, 0.6702008247375488, 0.6643370985984802], 'accuracy': [0.49871110916137695, 0.5064444541931152, 0.5264000296592712, 0.5385333299636841, 0.5563555359840393, 0.5614666938781738, 0.5766666531562805, 0.5895110964775085, 0.6000000238418579, 0.6095555424690247, 0.6185333132743835, 0.6323555707931519, 0.6407999992370605, 0.647599995136261, 0.6585777997970581, 0.6676889061927795, 0.6765778064727783, 0.6834222078323364], 'val_loss': [0.731362521648407, 0.7148647904396057, 0.7000304460525513, 0.6839893460273743, 0.6753506064414978, 0.6637153625488281, 0.6567765474319458, 0.6463953852653503, 0.6357491612434387, 0.6274867057800293, 0.6196037530899048, 0.6126242280006409, 0.6046810746192932, 0.5980660915374756, 0.590559184551239, 0.582603931427002, 0.5741293430328369, 0.5667080283164978], 'val_accuracy': [0.5055999755859375, 0.5332000255584717, 0.5600000023841858, 0.590399980545044, 0.6064000129699707, 0.6348000168800354, 0.6460000276565552, 0.6632000207901001, 0.6772000193595886, 0.6931999921798706, 0.6995999813079834, 0.7075999975204468, 0.7196000218391418, 0.7268000245094299, 0.734000027179718, 0.7432000041007996, 0.7523999810218811, 0.7580000162124634]}\n"
     ]
    }
   ],
   "source": [
    "#Model Created with Keras\n",
    "model = Sequential()\n",
    "model.add(Dense(512, activation='relu', input_dim=NUM_WORDS))\n",
    "model.add(Dropout(0.3))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(2, activation='sigmoid'))\n",
    "opt = keras.optimizers.Adam(learning_rate=0.00001)\n",
    "\n",
    "model.compile(\n",
    " optimizer = opt,\n",
    " loss = \"binary_crossentropy\",\n",
    " metrics = [\"accuracy\"]\n",
    ")\n",
    "\n",
    "\n",
    "results = model.fit(\n",
    " x_train, y_train,\n",
    " epochs= 18,\n",
    " batch_size = 512,\n",
    " validation_data = (x_test, y_test)\n",
    ")\n",
    "\n",
    "\n",
    "print(results.history)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N5OiEIQFSqCR"
   },
   "source": [
    "## Attacking\n",
    "\n",
    "With our model trained, we can create a  `ModelWrapper` that will allow us to run TextAttack on a custom Keras model. Each `ModelWrapper` must implement a single method, `__call__`, which takes a list of strings and returns a `List`, `np.ndarray`, or `torch.Tensor` of predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W61gKHFs6Wj0",
    "outputId": "a6b1bd94-e9cf-42b3-af2d-086b4b4cb5de"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.44404104, 0.5262513 ],\n",
       "       [0.49010894, 0.49974558]], dtype=float32)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class CustomKerasModelWrapper(ModelWrapper):\n",
    "    def __init__(self, model):\n",
    "        self.model = model\n",
    "\n",
    "    def __call__(self, text_input_list):\n",
    "      \n",
    "      x_transform = []\n",
    "      for i, review in enumerate(text_input_list):\n",
    "        tokens = [x.strip(\",\") for x in review.split()]\n",
    "        BoW_array = np.zeros((NUM_WORDS,))\n",
    "        for word in tokens:\n",
    "          if word in vocabulary:\n",
    "            if vocabulary[word] < len(BoW_array):\n",
    "              BoW_array[vocabulary[word]] += 1            \n",
    "        x_transform.append(BoW_array)\n",
    "      x_transform = np.array(x_transform)\n",
    "      prediction = self.model.predict(x_transform)\n",
    "      return prediction\n",
    "\n",
    "\n",
    "CustomKerasModelWrapper(model)([\"bad bad bad bad bad\", \"good good good good\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wz7MCIx9THHT"
   },
   "source": [
    "With our `ModelWrapper` constructed, we can use TextAttack's HuggingFaceDataset module to load reviews for testing, alongside TextAttack's PWWSRen2019 module to serve as our attack recipe. \n",
    "\n",
    "The attack below leverages TextAttack's `Attack` class, capable of running attacks against entire datasets. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "2c5338fb566b47ddbc062784039754f7",
      "3763a1ad84654e058109457844f5a0b9",
      "bccc1e8942d448e7bfc2c21ba02867cc",
      "0566102bb2624dd9acf93933eae23df1",
      "efcfc1d8b2ae4caa85eae185d30d0083",
      "fb72fd87d4f74fc2a4ea82c9b7fd5206",
      "0f12e7ddd8de407a8fcde5cdcd8610ad",
      "05986d4a9f464139b53472e18f0fc502",
      "53a8ab54de1642e89580289b20e4c26c",
      "85461fc66e4645a1bd339ee7f3ff3771",
      "218e0c96d70a46468530ebeb59952eba",
      "2a1a0b105421491aad6c008f663e5feb",
      "6d5fc859d79d4f2582f383dec98dad11",
      "587eb79f4d8e48dda4568b5d7ac9c637",
      "b7cc3a695be146b78dab880d1ae0330d",
      "538730dd801a4bfd87a7707ca8cd5f0a",
      "7ac7e3137112454a8f7232fbf523ab67",
      "66360e9bacd94b3aa8ec697e7cee1c0c",
      "c18fb50fa3834c64a7c1626f9c4f6304",
      "082ec92f7642468d9d482132b2c65816",
      "e5773a8588614606bbce8c56b0ec6de9",
      "6fafbace6e43458f838aebdb4f45d93e",
      "30ad2aae21ce46db9df10683a0a2b0e9",
      "7602a005fba545318811c4d63af70ba2",
      "27141b5bfff74e989ab44026865a0956",
      "0062a511ed7442749010b67572820556",
      "e3ce7e0bc05e4b53913a90fe5a7272fa",
      "83d9e1f8629a4a4b84dfab02078fb206",
      "ffa4e17dfc07403ba2cdc7465a928a31",
      "c84995270b8a4ae7995b6f03d3a4f378",
      "4edc4abd1091410ebd8df725e14631f4",
      "39ad0852b0394fc2834e0975efc5ee30",
      "c61d7f75c03640c4aff74a7aaf275f37",
      "8406bfbc47394297842de58841cd204b",
      "f2fed3231b944f15ad31b17305340520",
      "3c92027dbe164efc9dec3e1d5e15b801",
      "71d45357135c482abd427737bcb0d480",
      "307365d9931f49bc925c1ef2f5e2c46a",
      "1ac917843df14def88cc50c0e57e6bd3",
      "356963f2891b42278b0bd386a6159f7f",
      "79ba1a9f5a5e4e029926b66d5be576bb",
      "e721fa08373e44cca0ba89521b442fee",
      "48c3c238633c4539aa97d31686e89440",
      "c2cfe61dfb644f22ad2e53f82993b136",
      "63c1d88755cd445a891c68598045d846",
      "0b719255719744dcb8067ac2a32c5a37",
      "b91f2e95fdb84c15a5b336ec58cfb592",
      "f7b40ae27c0941e6844678278566c6f5"
     ]
    },
    "id": "1NQusSfN40aK",
    "outputId": "ef1de863-a4e7-414d-d7b0-17d77df314e9"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default\n",
      "Reusing dataset rotten_tomatoes_movie_review (/p/qdata/jy2ma/.cache/textattack/datasets/rotten_tomatoes_movie_review/default/1.0.0/9c411f7ecd9f3045389de0d9ce984061a1056507703d2e3183b1ac1a90816e4d)\n",
      "textattack: Loading \u001b[94mdatasets\u001b[0m dataset \u001b[94mrotten_tomatoes\u001b[0m, split \u001b[94mtest\u001b[0m.\n",
      "textattack: Unknown if model of class <class 'tensorflow.python.keras.engine.sequential.Sequential'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n",
      "[Succeeded / Failed / Skipped / Total] 0 / 0 / 1 / 1:  10%|█         | 1/10 [00:00<00:00, 17.58it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attack(\n",
      "  (search_method): GreedyWordSwapWIR(\n",
      "    (wir_method):  weighted-saliency\n",
      "  )\n",
      "  (goal_function):  UntargetedClassification\n",
      "  (transformation):  WordSwapWordNet\n",
      "  (constraints): \n",
      "    (0): RepeatModification\n",
      "    (1): StopwordModification\n",
      "  (is_black_box):  True\n",
      ") \n",
      "\n",
      "--------------------------------------------- Result 1 ---------------------------------------------\n",
      "\u001b[91mNegative (50%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
      "\n",
      "lovingly photographed in the manner of a golden book sprung to life , stuart little 2 manages sweetness largely without stickiness .\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 0 / 1 / 1 / 2:  20%|██        | 2/10 [00:00<00:00,  8.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 2 ---------------------------------------------\n",
      "\u001b[92mPositive (50%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "consistently clever and suspenseful .\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 1 / 1 / 3 / 5:  50%|█████     | 5/10 [00:00<00:00,  5.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 3 ---------------------------------------------\n",
      "\u001b[92mPositive (50%)\u001b[0m --> \u001b[91mNegative (50%)\u001b[0m\n",
      "\n",
      "it's \u001b[92mlike\u001b[0m a \" big chill \" reunion of the baader-meinhof \u001b[92mgang\u001b[0m , only these guys are more harmless pranksters than political activists .\n",
      "\n",
      "it's \u001b[91msimilar\u001b[0m a \" big chill \" reunion of the baader-meinhof \u001b[91mbunch\u001b[0m , only these guys are more harmless pranksters than political activists .\n",
      "\n",
      "\n",
      "--------------------------------------------- Result 4 ---------------------------------------------\n",
      "\u001b[91mNegative (51%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
      "\n",
      "the story gives ample opportunity for large-scale action and suspense , which director shekhar kapur supplies with tremendous skill .\n",
      "\n",
      "\n",
      "--------------------------------------------- Result 5 ---------------------------------------------\n",
      "\u001b[91mNegative (50%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
      "\n",
      "red dragon \" never cuts corners .\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 2 / 1 / 5 / 8:  80%|████████  | 8/10 [00:01<00:00,  6.08it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 6 ---------------------------------------------\n",
      "\u001b[92mPositive (50%)\u001b[0m --> \u001b[91mNegative (51%)\u001b[0m\n",
      "\n",
      "fresnadillo has something serious to \u001b[92msay\u001b[0m about the ways in which extravagant chance can distort our perspective and throw us off the path of good sense .\n",
      "\n",
      "fresnadillo has something serious to \u001b[91mtell\u001b[0m about the ways in which extravagant chance can distort our perspective and throw us off the path of good sense .\n",
      "\n",
      "\n",
      "--------------------------------------------- Result 7 ---------------------------------------------\n",
      "\u001b[91mNegative (51%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
      "\n",
      "throws in enough clever and unexpected twists to make the formula feel fresh .\n",
      "\n",
      "\n",
      "--------------------------------------------- Result 8 ---------------------------------------------\n",
      "\u001b[91mNegative (51%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
      "\n",
      "weighty and ponderous but every bit as filling as the treat of the title .\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 3 / 1 / 5 / 9:  90%|█████████ | 9/10 [00:01<00:00,  4.89it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 9 ---------------------------------------------\n",
      "\u001b[92mPositive (50%)\u001b[0m --> \u001b[91mNegative (50%)\u001b[0m\n",
      "\n",
      "a real audience-pleaser that will strike a chord with anyone who's ever waited in a doctor's office , emergency room , hospital bed or insurance \u001b[92mcompany\u001b[0m office .\n",
      "\n",
      "a real audience-pleaser that will strike a chord with anyone who's ever waited in a doctor's office , emergency room , hospital bed or insurance \u001b[91msociety\u001b[0m office .\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 4 / 1 / 5 / 10: 100%|██████████| 10/10 [00:02<00:00,  4.86it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 10 ---------------------------------------------\n",
      "\u001b[92mPositive (51%)\u001b[0m --> \u001b[91mNegative (50%)\u001b[0m\n",
      "\n",
      "generates an enormous \u001b[92mfeeling\u001b[0m of empathy for its characters .\n",
      "\n",
      "generates an enormous \u001b[91mlook\u001b[0m of empathy for its characters .\n",
      "\n",
      "\n",
      "\n",
      "+-------------------------------+-------+\n",
      "| Attack Results                |       |\n",
      "+-------------------------------+-------+\n",
      "| Number of successful attacks: | 4     |\n",
      "| Number of failed attacks:     | 1     |\n",
      "| Number of skipped attacks:    | 5     |\n",
      "| Original accuracy:            | 50.0% |\n",
      "| Accuracy under attack:        | 10.0% |\n",
      "| Attack success rate:          | 80.0% |\n",
      "| Average perturbed word %:     | 7.24% |\n",
      "| Average num. words per input: | 15.4  |\n",
      "| Avg num queries:              | 103.2 |\n",
      "+-------------------------------+-------+\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<textattack.attack_results.skipped_attack_result.SkippedAttackResult at 0x7f2d494c67f0>,\n",
       " <textattack.attack_results.failed_attack_result.FailedAttackResult at 0x7f2d40ab9520>,\n",
       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f2d46675be0>,\n",
       " <textattack.attack_results.skipped_attack_result.SkippedAttackResult at 0x7f2d4740da60>,\n",
       " <textattack.attack_results.skipped_attack_result.SkippedAttackResult at 0x7f2d40aca130>,\n",
       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f2d4289e9d0>,\n",
       " <textattack.attack_results.skipped_attack_result.SkippedAttackResult at 0x7f2d42fa9820>,\n",
       " <textattack.attack_results.skipped_attack_result.SkippedAttackResult at 0x7f2d3b54eb50>,\n",
       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f2d4905ce50>,\n",
       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f2d49032940>]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textattack import AttackArgs\n",
    "from textattack.datasets import Dataset\n",
    "from textattack import Attacker\n",
    "\n",
    "model_wrapper = CustomKerasModelWrapper(model)\n",
    "dataset = HuggingFaceDataset(\"rotten_tomatoes\", None, \"test\", shuffle=True)\n",
    "\n",
    "attack = PWWSRen2019.build(model_wrapper)\n",
    "\n",
    "attack_args = AttackArgs(num_examples=10, checkpoint_dir=\"checkpoints\")\n",
    "\n",
    "attacker = Attacker(attack, dataset, attack_args)\n",
    "\n",
    "attacker.attack_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "se8V4zjqUiji"
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "Great! We trained a binary classifier, created a custom `ModelWrapper` for Keras models, and successsfully ran adversarial attacks against our trained Keras model! This serves a basic demo for how to use TextAttack within your own environments. \n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "text_attack_keras_example_fixed_version",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0062a511ed7442749010b67572820556": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0566102bb2624dd9acf93933eae23df1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_05986d4a9f464139b53472e18f0fc502",
      "placeholder": "​",
      "style": "IPY_MODEL_0f12e7ddd8de407a8fcde5cdcd8610ad",
      "value": " 4.97k/? [00:00&lt;00:00, 13.3kB/s]"
     }
    },
    "05986d4a9f464139b53472e18f0fc502": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "082ec92f7642468d9d482132b2c65816": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7602a005fba545318811c4d63af70ba2",
      "placeholder": "​",
      "style": "IPY_MODEL_30ad2aae21ce46db9df10683a0a2b0e9",
      "value": " 488k/488k [00:01&lt;00:00, 396kB/s]"
     }
    },
    "0b719255719744dcb8067ac2a32c5a37": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0f12e7ddd8de407a8fcde5cdcd8610ad": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1ac917843df14def88cc50c0e57e6bd3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "218e0c96d70a46468530ebeb59952eba": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: ",
      "description_tooltip": null,
      "layout": "IPY_MODEL_587eb79f4d8e48dda4568b5d7ac9c637",
      "max": 869,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_6d5fc859d79d4f2582f383dec98dad11",
      "value": 869
     }
    },
    "27141b5bfff74e989ab44026865a0956": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e3ce7e0bc05e4b53913a90fe5a7272fa",
       "IPY_MODEL_83d9e1f8629a4a4b84dfab02078fb206"
      ],
      "layout": "IPY_MODEL_0062a511ed7442749010b67572820556"
     }
    },
    "2a1a0b105421491aad6c008f663e5feb": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_538730dd801a4bfd87a7707ca8cd5f0a",
      "placeholder": "​",
      "style": "IPY_MODEL_b7cc3a695be146b78dab880d1ae0330d",
      "value": " 1.88k/? [00:00&lt;00:00, 30.7kB/s]"
     }
    },
    "2c5338fb566b47ddbc062784039754f7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_bccc1e8942d448e7bfc2c21ba02867cc",
       "IPY_MODEL_0566102bb2624dd9acf93933eae23df1"
      ],
      "layout": "IPY_MODEL_3763a1ad84654e058109457844f5a0b9"
     }
    },
    "307365d9931f49bc925c1ef2f5e2c46a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "30ad2aae21ce46db9df10683a0a2b0e9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "356963f2891b42278b0bd386a6159f7f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3763a1ad84654e058109457844f5a0b9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "39ad0852b0394fc2834e0975efc5ee30": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3c92027dbe164efc9dec3e1d5e15b801": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_356963f2891b42278b0bd386a6159f7f",
      "placeholder": "​",
      "style": "IPY_MODEL_1ac917843df14def88cc50c0e57e6bd3",
      "value": " 1066/0 [00:00&lt;00:00, 8169.70 examples/s]"
     }
    },
    "48c3c238633c4539aa97d31686e89440": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0b719255719744dcb8067ac2a32c5a37",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_63c1d88755cd445a891c68598045d846",
      "value": 1
     }
    },
    "4edc4abd1091410ebd8df725e14631f4": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "538730dd801a4bfd87a7707ca8cd5f0a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "53a8ab54de1642e89580289b20e4c26c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_218e0c96d70a46468530ebeb59952eba",
       "IPY_MODEL_2a1a0b105421491aad6c008f663e5feb"
      ],
      "layout": "IPY_MODEL_85461fc66e4645a1bd339ee7f3ff3771"
     }
    },
    "587eb79f4d8e48dda4568b5d7ac9c637": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "63c1d88755cd445a891c68598045d846": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "66360e9bacd94b3aa8ec697e7cee1c0c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6d5fc859d79d4f2582f383dec98dad11": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "6fafbace6e43458f838aebdb4f45d93e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "71d45357135c482abd427737bcb0d480": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "7602a005fba545318811c4d63af70ba2": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "79ba1a9f5a5e4e029926b66d5be576bb": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_48c3c238633c4539aa97d31686e89440",
       "IPY_MODEL_c2cfe61dfb644f22ad2e53f82993b136"
      ],
      "layout": "IPY_MODEL_e721fa08373e44cca0ba89521b442fee"
     }
    },
    "7ac7e3137112454a8f7232fbf523ab67": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c18fb50fa3834c64a7c1626f9c4f6304",
       "IPY_MODEL_082ec92f7642468d9d482132b2c65816"
      ],
      "layout": "IPY_MODEL_66360e9bacd94b3aa8ec697e7cee1c0c"
     }
    },
    "83d9e1f8629a4a4b84dfab02078fb206": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_39ad0852b0394fc2834e0975efc5ee30",
      "placeholder": "​",
      "style": "IPY_MODEL_4edc4abd1091410ebd8df725e14631f4",
      "value": " 8530/0 [00:00&lt;00:00, 13612.90 examples/s]"
     }
    },
    "8406bfbc47394297842de58841cd204b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "85461fc66e4645a1bd339ee7f3ff3771": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b7cc3a695be146b78dab880d1ae0330d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b91f2e95fdb84c15a5b336ec58cfb592": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "bccc1e8942d448e7bfc2c21ba02867cc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: ",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fb72fd87d4f74fc2a4ea82c9b7fd5206",
      "max": 1860,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_efcfc1d8b2ae4caa85eae185d30d0083",
      "value": 1860
     }
    },
    "c18fb50fa3834c64a7c1626f9c4f6304": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6fafbace6e43458f838aebdb4f45d93e",
      "max": 487770,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e5773a8588614606bbce8c56b0ec6de9",
      "value": 487770
     }
    },
    "c2cfe61dfb644f22ad2e53f82993b136": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f7b40ae27c0941e6844678278566c6f5",
      "placeholder": "​",
      "style": "IPY_MODEL_b91f2e95fdb84c15a5b336ec58cfb592",
      "value": " 1066/0 [00:00&lt;00:00, 8779.95 examples/s]"
     }
    },
    "c61d7f75c03640c4aff74a7aaf275f37": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f2fed3231b944f15ad31b17305340520",
       "IPY_MODEL_3c92027dbe164efc9dec3e1d5e15b801"
      ],
      "layout": "IPY_MODEL_8406bfbc47394297842de58841cd204b"
     }
    },
    "c84995270b8a4ae7995b6f03d3a4f378": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e3ce7e0bc05e4b53913a90fe5a7272fa": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c84995270b8a4ae7995b6f03d3a4f378",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ffa4e17dfc07403ba2cdc7465a928a31",
      "value": 1
     }
    },
    "e5773a8588614606bbce8c56b0ec6de9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "e721fa08373e44cca0ba89521b442fee": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "efcfc1d8b2ae4caa85eae185d30d0083": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "f2fed3231b944f15ad31b17305340520": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_307365d9931f49bc925c1ef2f5e2c46a",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_71d45357135c482abd427737bcb0d480",
      "value": 1
     }
    },
    "f7b40ae27c0941e6844678278566c6f5": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fb72fd87d4f74fc2a4ea82c9b7fd5206": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ffa4e17dfc07403ba2cdc7465a928a31": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
